Benchmarking Data Curation Systems

Data curation includes the many tasks needed to ensure data maintains its value over time. Given the maturity of many data curation tasks, including data transformation and data cleaning, it is surprising that rigorous empirical evaluations of research ideas are so scarce. In this work, we argue that thorough evaluation of data curation systems imposes several major obstacles that need to be overcome. First, we consider the outputs generated by a data curation system (for example, an integrated or cleaned database or a set of constraints produced by a schema discovery system). To compare the results of different systems, measures of output quality should be agreed upon by the community and, since such measures can be quite complex, publicly available implementations of these measures should be developed, shared, and optimized. Second, we consider the inputs to the data curation system. New techniques are needed to generate and control the metadata and data that are the input to curation systems. For a thorough evaluation, it must be possible to control (and systematically vary) input characteristics such as the number of errors in data cleaning or the complexity of a schema mapping in data transformation. Finally, we consider benchmarks. Data and metadata generators must support the creation of reasonable goldstandard outputs for different curation tasks and must promote productivity by enabling the creation of a large number of inputs with little manual effort. In this work, we overview some recent advances in addressing these important obstacles. We argue that evaluation of curation systems is itself a fascinating and important research area and challenges the curation community to tackle some of the remaining open research problems.